Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

When bootstrapping an amazon elastic map reduce job, can my script use sudo?

I need to:

sudo apt-get install rubygems
sudo gem install <lots of gems>

does the bootstrap action have sudo access?

like image 758
Matthew Rathbone Avatar asked Dec 02 '10 15:12

Matthew Rathbone


People also ask

When can bootstrap actions scripts can be defined in EMR?

Bootstrap actions are scripts that run on cluster after Amazon EMR launches the instance using the Amazon Linux Amazon Machine Image (AMI). Bootstrap actions run before Amazon EMR installs the applications that you specify when you create the cluster and before cluster nodes begin processing data.

How is Amazon Elastic MapReduce?

Amazon EMR (previously called Amazon Elastic MapReduce) is a managed cluster platform that simplifies running big data frameworks, such as Apache Hadoop and Apache Spark , on AWS to process and analyze vast amounts of data.

What is AWS bootstrapping?

Bootstrapping is the deployment of a AWS CloudFormation template to a specific AWS environment (account and region). The bootstrapping template accepts parameters that customize some aspects of the bootstrapped resources (see Customizing bootstrapping).

What is bootstrapping in Hadoop?

Bootstrap actions are scripts that run as the Hadoop user by default—but they can also run as the root user with the sudo command. You can configure bootstrap actions to run commands conditionally, based on instance-specific values in the instance. json or job-flow.


1 Answers

The answer is yes. You can test your bootstrap script like this:

elastic_mapreduce --create --alive --ssh

This will create a node and give you a ssh connection to it, from which you can test your bootstrap script.

UPDATE: For reference here is what I'm running:

#!/bin/bash
sudo apt-get -y -V install irb1.8 libreadline-ruby1.8 libruby libruby1.8 rdoc1.8 ruby ruby1.8 ruby1.8-dev
wget http://production.cf.rubygems.org/rubygems/rubygems-1.8.11.zip
unzip rubygems-1.8.11.zip
cd rubygems-1.8.11
sudo ruby setup.rb
sudo gem1.8 install bson bson_ext json tzinfo i18n activesupport --no-rdoc --no-ri

UPDATE2: to install aws-sdk

#!/bin/bash

# ruby developer packages
sudo apt-get -y -V  install ruby1.8-dev ruby1.8 ri1.8 rdoc1.8 irb1.8
sudo apt-get -y -V  install libreadline-ruby1.8 libruby1.8 libopenssl-ruby
# nokogiri requirements
sudo apt-get -y -V  install libxslt-dev libxml2-dev

wget http://production.cf.rubygems.org/rubygems/rubygems-1.8.11.zip
unzip rubygems-1.8.11.zip
cd rubygems-1.8.11
sudo ruby setup.rb

sudo gem1.8 install aws-sdk --no-rdoc --no-ri

-y on apt-get makes it not prompt you

I wget rubygems because the version you get with apt-get is way out of date, and some gems won't build using an old version.

like image 181
Matthew Rathbone Avatar answered Oct 04 '22 11:10

Matthew Rathbone