How to run 2 EMR Spark Step Concurrently?

Question

I am trying to have 2 steps run concurrent in EMR. However I always get the first step running and the second pending.

Part of my Yarn configuration is as follows:

{
    "Classification": "capacity-scheduler",
    "Properties": {
    "yarn.scheduler.capacity.resource-calculator": "org.apache.hadoop.yarn.util.resource.DominantResourceCalculator",
    "yarn.scheduler.capacity.maximum-am-resource-percent": "0.5"
    }
  }

When I run on my local Mac I am able to run the 2 application on Yarn with similar configuration, where the change are actually spark submit resource request, to match the cluster capacity and performance required.

In other words, My yarn is set up to run multiple application.

Hence, before i dig hard into it, i wonder if it is actually possible to have the step run concurrently or only serially ?

Else is there any tips or something specific to run to job concurrently ?

My cluster is over capacitated with respect to what each job request. Hence i don't not understand why it can't run concurrently.

Answer 1

• Is it possible to have the step run concurrently or only serially?

  • Confirmed from AWS support people that we can not run multiple steps in parallel(concurrent), the steps are serial, so what you are seeing (ie second job in pending state) is expected.

• Is there any tips or something specific to run to job concurrently?

  • You can simply put both the spark-submit in a bash script and run the bash script, but you might loose some direct debugging info on the AWS web console (which imo is slow already), you can see these debugging info on the spark-history server

On your local mac, you are able to run multiple YARN application in parallel because you are submitting the applications to yarn directly, whereas in EMR the yarn/spark applications are submitted through AWS's internal `command-runner.jar`, it does a bunch of other logging/bootstrapping etc to be able to see the `emr step` info on the web console.