Menu
Following the steps of Sparkling Water from the link http://h2o-release.s3.amazonaws.com/sparkling-water/rel-2.2/0/index.html.
Running in terminal :
~/InstallFile/SparklingWater/sparkling-water-2.2.0$ bin/sparkling-shell --conf "spark.executor.memory=1g"
Please setup SPARK_HOME variable to your Spark installation
736
The SPARK_HOME variable is the directory/folder where sparkling water will find the spark run time.
Spark Configuration Spark properties control most application parameters and can be set by using a SparkConf object, or through Java system properties. Environment variables can be used to set per-machine settings, such as the IP address, through the conf/spark-env.sh script on each node.
Before starting PySpark, you need to set the following environments to set the Spark path and the Py4j path. Or, to set the above environments globally, put them in the . bashrc file. Then run the following command for the environments to work.
Enter the path to the Spark folder C:\Spark\spark-2.4. 5-bin-hadoop2. 7\bin. We recommend using %SPARK_HOME%\bin to avoid possible issues with the path.
You should install and set the SPARK_HOME variable, in unix terminal run the following code to set the variable:
export SPARK_HOME="/path/to/spark"
To maintain this config you should append this to the end of your .bashrc.
See this for installation https://www.tutorialspoint.com/apache_spark/apache_spark_installation.htm
51
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
