pyspark: Could not find valid SPARK_HOME

Question

After a fresh installation of pyspark via pip install pyspark I get the following errors:

> pyspark
Could not find valid SPARK_HOME while searching ['/Users', '/usr/local/bin']
/usr/local/bin/pyspark: line 24: /bin/load-spark-env.sh: No such file or directory
/usr/local/bin/pyspark: line 77: /bin/spark-submit: No such file or directory
/usr/local/bin/pyspark: line 77: exec: /bin/spark-submit: cannot execute: No such file or directory
> spark-shell
Could not find valid SPARK_HOME while searching ['/Users', '/usr/local/bin']
/usr/local/bin/spark-shell: line 57: /bin/spark-submit: No such file or directory

What is a valid SPARK_HOME, how do I set it, and why is there no default that works?

I've seen instructions on how to manually set the environment variable after a manual installation of spark, but I am wondering how to set it in this case after using pip to install pyspark.

I've installed only spark via brew install apache-spark, and the spark-shell that comes out of that installation works just fine out of the box. After installing pyspark after that, I get the messages above. Confusing.

Answer 1

> brew install apache-spark

actually already provides a working pyspark shell. It is not necessary to additionally

> pip install pyspark

and in fact this breaks the installation for me.