pyspark: pip install couldn't find a version

Question

I am trying to install the pyspark using pip install like below. But I got the following errors.

(python_virenv)edamame$ pip install pyspark
Collecting pyspark
  Could not find a version that satisfies the requirement pyspark (from versions: )
No matching distribution found for pyspark

Does anyone have any idea? Thanks!

Answer 1

As of Spark 2.2, PySpark is now available in PyPI.

pip install pyspark

---

As of Spark 2.1, PySpark is pip installable but not yet from PyPI, which is under consideration for 2.2 in this ticket. To install PySpark you now just need download Spark 2.1+ and run setup.py:

cd spark-2.1/python/
pip install -e .

Big thanks to @Holden!

Answer 2

pyspark is not in PyPI so you could not directly use pip install to install it.

Instead you could download a proper version of Spark here: http://spark.apache.org/downloads.html, and you will get a compressed TAR file. Then unpack it and pyspark is in its python folder.

To open the Python version of the Spark shell, you could go into your Spark directory and type:

bin/pyspark

or

bin\pyspark

in Windows.

Answer 3

pyspark doesn't even exist in PyPI as you can see from https://pypi.python.org/pypi?%3Aaction=search&term=pyspark&submit=search, that's why pip is telling you it can't find it.