What is the correct syntax for running a bash script as a step in EMR?

Question

I am trying to run a bash script as a step after EMR completes bootstrapping. Following is my terraform code:

step { action_on_failure = "CONTINUE" 
name = "Setup Hadoop configuration" 
hadoop_jar_step { 
jar = "command-runner.jar" 
args = ["bash,-c,'cd /mnt; chmod +x ./userdata.sh; ./userdata.sh'"] 
}}

This isn't working and keeps on failing mentioning that No file found. Are args correctly provided ? What about script-runner.jar? Any help is highly appreciated. :(

Answer 1

With command-runner.jar you can execute many programs like bash script, and you do not have to know its full path as was the case with script-runner.jar. It is recommended to use command-runner.jar.

EMR is in cluster mode, you do not know which nodes execute the shell script, so push it to S3 :

{
  "Name": "Setup Hadoop configuration",
  "ActionOnFailure": "CONTINUE",
 "HadoopJarStep": {
    "Jar": "command-runner.jar",
    "Args": [
         "bash",
         "-c",
         " aws s3 cp s3://path_to_bucket_S3/userdata.sh .;
          chmod +x userdata.sh;
          ./userdata.sh args...;
          rm userdata.sh "
     ]
  }
}