Spark executor on yarn-client does not take executor core count configuration.

Question

Irrespective of the spark executor core count, yarn container for the executor does not use more than 1 core.

Answer 1

YARN is showing 1 core per executor irrespective of spark.executor.cores because by default DefaultResourceCalculator is used. It considers only memory.

public int computeAvailableContainers(Resource available, Resource required) {
// Only consider memory
return available.getMemory() / required.getMemory();
  }

Use DominantResourceCalculator, It uses both cpu and memory.

Set below config in capacity-scheduler.xml

yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator

More about DominantResourceCalculator