How to run Spark processing in parallel in Eclipse?

Question

I would like to run a Spark application using multiple executors in parallel while trying the application on my development environment (Eclipse). It seems the Spark engine serializes all the tasks and run them using one executor.

Is there an option to run two or more tasks in parallel in Eclipse with spark.master=local?

Answer 1

Use spark.master="local[n]" where n is the number of cores you want to assign to spark or "*" for all cores.