Submitting pyspark script to a remote Spark server?

Question

This is probably a really silly question, but I can't find the answer with Google. I've written a simple pyspark ETL script that reads in a CSV and writes it to Parquet, something like this:

spark = SparkSession.builder.getOrCreate()
sqlContext = SQLContext(spark.sparkContext)
df = sqlContext.read.csv(input_filename)
df.write.parquet(output_path)

To run it, I start up a local Spark cluster in Docker:

$ docker run --network=host jupyter/pyspark-notebook

I run the Python script and it connects to this local Spark cluster and all works as expected.

Now I'd like to run the same script on a remote Spark cluster (AWS EMR). Can I just specify a remote IP address somewhere when initialising the Spark context? Or am I misunderstanding how Spark works?

Answer 1

You can create a spark session by specifying the IP address of the remote master.

spark = SparkSession.builder.master("spark://<ip>:<port>").getOrCreate()

In case of AWS EMR, standalone mode is not supported. You need to use yarn in either client or cluster mode, and point HADOOP_CONF_DIR to a location on your local server where all files from /etc/hadoop/conf are present. Then setup dynamic port forwarding to connect to the EMR cluster. Create a spark session like:

spark = SparkSession.builder.master('yarn').config('spark.submit.deployMode', 'cluster').getOrCreate()

refer https://aws.amazon.com/premiumsupport/knowledge-center/emr-submit-spark-job-remote-cluster/