Apache Spark upgrade from 1.5.2 to 1.6.0 using homebrew leading to permission denied error during execution

Question

I just upgraded Spark from 1.5.2 to 1.6.0 using Homebrew and reset my SPARK_HOME environment variable to /usr/local/Cellar/apache-spark/1.6.0. Now while executing pyspark, it gives a permission denied error.

If I go into the earlier 1.5.2 installation directory and execute pyspark from there, it runs fine. But running pyspark from the 1.6.0 installation directory fails with this permission denied error.

/usr/local/Cellar/apache-spark/1.6.0/bin/load-spark-env.sh: line 2: /usr/local/Cellar/apache-spark/1.6.0/libexec/bin/load-spark-env.sh: Permission denied

/usr/local/Cellar/apache-spark/1.6.0/bin/load-spark-env.sh: line 2: exec: /usr/local/Cellar/apache-spark/1.6.0/libexec/bin/load-spark-env.sh: cannot execute: Undefined error: 0

What could be causing this?

Answer 1

I ran into the same issue and the easiest fix is to set $SPARK_HOME to /usr/local/Cellar/apache-spark/<your_spark_version>/libexec/ instead.

You can also build from source directly and you can find the instructions here.

Basically just do

git clone https://github.com/apache/spark/`
cd spark
git checkout origin/branch-X.Y

build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package

You'll need to set the $SPARK_HOME to the top level directory of the spark source code.

Answer 2

For Pyspark 2.4.4 in MacOs add to .bash_profile:

# Pyspark
export SPARK_HOME='/usr/local/Cellar/apache-spark/2.4.4/libexec/'
export PYSPARK_DRIVER_PYTHON="jupyter"
export PYSPARK_DRIVER_PYTHON_OPTS="notebook"
#For python 3, You have to add the line below or you will get an 
# error
export PYSPARK_PYTHON=python3
alias snotebook='$SPARK_PATH/bin/pyspark --master local[4]'

then reload with

source .bash_profile

and start pyspark with

pyspark 

Do not unset SPARK_HOME or you will have the following error:

/Users/user/opt/anaconda3/bin/pyspark: line 24: /bin/load-spark-env.sh: No such file or directory
/Users/user/opt/anaconda3/bin/pyspark: line 77: /bin/spark-submit: No such file or directory
/Users/usr/opt/anaconda3/bin/pyspark: line 77: exec: /bin/spark-submit: cannot execute: No such file or directory