How to bootstrap installation of Python modules on Amazon EMR?

Question

I want to do something really basic, simply fire up a Spark cluster through the EMR console and run a Spark script that depends on a Python package (for example, Arrow). What is the most straightforward way of doing this?

Answer 1

The most straightforward way would be to create a bash script containing your installation commands, copy it to S3, and set a bootstrap action from the console to point to your script.

Here's an example I'm using in production:

s3://mybucket/bootstrap/install_python_modules.sh

#!/bin/bash -xe  # Non-standard and non-Amazon Machine Image Python modules: sudo pip install -U \   awscli            \   boto              \   ciso8601          \   ujson             \   workalendar  sudo yum install -y python-psycopg2 

Answer 2

In short, there are two ways to install packages with pip, depending on the platform. First, you install whatever you need and then you can run your Spark step. The easiest is to use emr-4.0.0 and 'command-runner.jar':

from boto.emr.step import JarStep >>> pip_step=JarStep(name="Command Runner", ...             jar="command-runner.jar", ...             action_on_failure="CONTINUE", ...             step_args=['sudo','pip','install','arrow'] ... ) >>> spark_step=JarStep(name="Spark with Command Runner", ...                    jar="command-runner.jar", ...                    step_args=["spark-submit","/usr/lib/spark/examples/src/main/python/pi.py"] ...                    action_on_failure="CONTINUE" ) >>> step_list=conn.add_jobflow_steps(emr.jobflowid, [pip_step,spark_step]) 

On 2.x and 3.x, you use script-runner.jar in a similar fashion except that you have to specify the full URI for scriptrunner.

EDIT: Sorry, I didn't see that you wanted to do this through console. You can add the same steps in the console as well. The first step would be a Customer JAR with the same args as above. The second step is a spark step. Hope this helps!