importing pyspark in python shell

Question

This is a copy of someone else's question on another forum that was never answered, so I thought I'd re-ask it here, as I have the same issue. (See http://geekple.com/blogs/feeds/Xgzu7/posts/351703064084736)

I have Spark installed properly on my machine and am able to run python programs with the pyspark modules without error when using ./bin/pyspark as my python interpreter.

However, when I attempt to run the regular Python shell, when I try to import pyspark modules I get this error:

from pyspark import SparkContext 

and it says

"No module named pyspark". 

How can I fix this? Is there an environment variable I need to set to point Python to the pyspark headers/libraries/etc.? If my spark installation is /spark/, which pyspark paths do I need to include? Or can pyspark programs only be run from the pyspark interpreter?

Answer 1

Assuming one of the following:

• Spark is downloaded on your system and you have an environment variable SPARK_HOME pointing to it
• You have ran pip install pyspark

Here is a simple method (If you don't bother about how it works!!!)

Use findspark

1. Go to your python shell

   pip install findspark  import findspark findspark.init() 

2. import the necessary modules

   from pyspark import SparkContext from pyspark import SparkConf 

3. Done!!!