I need to:
sudo apt-get install rubygems
sudo gem install <lots of gems>
does the bootstrap action have sudo access?
Bootstrap actions are scripts that run on cluster after Amazon EMR launches the instance using the Amazon Linux Amazon Machine Image (AMI). Bootstrap actions run before Amazon EMR installs the applications that you specify when you create the cluster and before cluster nodes begin processing data.
Amazon EMR (previously called Amazon Elastic MapReduce) is a managed cluster platform that simplifies running big data frameworks, such as Apache Hadoop and Apache Spark , on AWS to process and analyze vast amounts of data.
Bootstrapping is the deployment of a AWS CloudFormation template to a specific AWS environment (account and region). The bootstrapping template accepts parameters that customize some aspects of the bootstrapped resources (see Customizing bootstrapping).
Bootstrap actions are scripts that run as the Hadoop user by default—but they can also run as the root user with the sudo command. You can configure bootstrap actions to run commands conditionally, based on instance-specific values in the instance. json or job-flow.
The answer is yes. You can test your bootstrap script like this:
elastic_mapreduce --create --alive --ssh
This will create a node and give you a ssh connection to it, from which you can test your bootstrap script.
UPDATE: For reference here is what I'm running:
#!/bin/bash
sudo apt-get -y -V install irb1.8 libreadline-ruby1.8 libruby libruby1.8 rdoc1.8 ruby ruby1.8 ruby1.8-dev
wget http://production.cf.rubygems.org/rubygems/rubygems-1.8.11.zip
unzip rubygems-1.8.11.zip
cd rubygems-1.8.11
sudo ruby setup.rb
sudo gem1.8 install bson bson_ext json tzinfo i18n activesupport --no-rdoc --no-ri
UPDATE2: to install aws-sdk
#!/bin/bash
# ruby developer packages
sudo apt-get -y -V install ruby1.8-dev ruby1.8 ri1.8 rdoc1.8 irb1.8
sudo apt-get -y -V install libreadline-ruby1.8 libruby1.8 libopenssl-ruby
# nokogiri requirements
sudo apt-get -y -V install libxslt-dev libxml2-dev
wget http://production.cf.rubygems.org/rubygems/rubygems-1.8.11.zip
unzip rubygems-1.8.11.zip
cd rubygems-1.8.11
sudo ruby setup.rb
sudo gem1.8 install aws-sdk --no-rdoc --no-ri
-y on apt-get makes it not prompt you
I wget rubygems because the version you get with apt-get is way out of date, and some gems won't build using an old version.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With