Installing Apache Spark on Windows

Question

I need to install Apache Spark on a Windows machine. According to the documentation I should have sbt installed on my machine and also override its default options to use a maximum of 2G of RAM.

After trying a lot I decided to choose maven instead (I overrided the default options to use 512MB) and I could run mvn compile that started to download some dependencies until it reached Zinc server. It said Zinc server didn`t run on port 8080 and maven failed the build.

Do I need to install the Zinc server locally and add it to Windows path? Can I compile my project without the server?

Answer 1

I've just started learning Apache Spark so I might be far from the valid answer. It worked for me on Mac OS X so your millage may vary.

The build process is described in Building:

Spark uses Simple Build Tool, which is bundled with it. To compile the code, go into the top-level Spark directory and run

sbt/sbt assembly

I've not changed the default build settings to do the assembly.

Yesterday's build from the sources:

jacek:~/oss/spark
$ sbt/sbt assembly
...
[success] Total time: 3632 s, completed May 6, 2014 2:39:07 AM

Today's build from the sources with some environment variables that add few other modules to the final assembly:

jacek:~/oss/spark
$ SPARK_HADOOP_VERSION=2.4.0 SPARK_YARN=true SPARK_HIVE=true sbt/sbt assembly
...
[success] Total time: 4564 s, completed May 7, 2014 1:49:56 AM