Does a flatMap in spark cause a shuffle?

Question

Does flatMap in spark behave like the map function and therefore cause no shuffling, or does it trigger a shuffle. I suspect it does cause shuffling. Can someone confirm it?

Answer 1

There is no shuffling with either map or flatMap. The operations that cause shuffle are:

• Repartition operations:
  • Repartition:
  • Coalesce:
• ByKey operations (except for counting):
  • GroupByKey:
  • ReduceByKey:
• Join operations:
  • Cogroup:
  • Join:

Although the set of elements in each partition of newly shuffled data will be deterministic, and so is the ordering of partitions themselves, the ordering of these elements is not. If one desires predictably ordered data following shuffle then it’s possible to use:

• mapPartitions to sort each partition using, for example, .sorted
• repartitionAndSortWithinPartitions to efficiently sort partitions while simultaneously repartitioning
• sortBy to make a globally ordered RDD

More info here: http://spark.apache.org/docs/latest/programming-guide.html#shuffle-operations