Spark executor max memory limit

Question

Am wondering if there is any size limit to Spark executor memory ?

Considering the case of running a badass job doing collect, unions, count, etc.

Just a bit of context, let's say I have these resources (2 machines)

Cores: 40 cores, Total = 80 cores
Memory: 156G, Total = 312

What's the recommendation, bigger vs smaller executors ?

Answer 1

The suggestion by Spark development team is to not have an executor that is more than 64GB or so (often mentioned in training videos by Databricks). The idea is that a larger JVM will have a larger Heap that can result in really slow garbage collection cycles.

I think is a good practice to have your executors 32GB or even 24GB or 16GB. So instead of having one large one you have 2-4 smaller ones.

It will perhaps have some more coordination overhead, but I think these should be ok for the vast majority of applications.

If you have not read this post, please do.