What is the relationship between tasks and partitions?

Question

Can I say?

1. The number of the Spark tasks equal to the number of the Spark partitions?

2. The executor runs once (batch inside of executor) is equal to one task?

3. Every task produce only a partition?

4. (duplicate of 1.)

Answer 1

The degree of parallelism, or the number of tasks that can be ran concurrently, is set by:

• The number of Executor Instances (configuration)
• The Number of Cores per Executor (configuration)
• The Number of Partitions being used (coded)

Actual parallelism is the smaller of

• executors * cores - which gives the amount of slots available to run tasks
• partitions - each partition will translate to a task whenever a slot opens up.

Tasks that run on the same executor will share the same JVM. This is used by the Broadcast feature as you only need one copy of the Broadcast data per Executor for all tasks to be able to access it through shared memory.

You can have multiple executors running, on the same machine, or on different machines. Executors are the true means of scalability.

Note that each Task takes up one Thread ¹, and is assumed to be assigned to one core ².

So -

1. Is the number of the Spark tasks equal to the number of the Spark partitions?

No (see previous).

2. The executor runs once (batch inside of executor) is equal to one task?

An Executor is started as an environment for the tasks to run. Multiple tasks will run concurrently within that Executor (multiple threads).

3. Every task produce only a partition?

For a task, it is one Partition in, one partition out. However, a repartitioning or shuffle/sort can happen in between tasks.

4. The number of the Spark tasks equal to the number of the Spark partitions?

Same as (1.)

(¹) Assumption is that within your tasks, you are not multithreading yourself (never do that, otherwise core estimate will be off).

(²) Note that due to hyper-threading, you might have more than one virtual core per physical core, and thus you can have several threads per core. You might even be able to handle multiple threads (2 to 3) on a single core without hyper-threading.

Answer 2

Partitions are a feature of RDD and are only available at design time (before an action is called).

Tasks are part of TaskSet per Stage per ActiveJob in a Spark application.

Is the number of the Spark tasks equal to the number of the Spark partitions?

Yes.

The executor runs once (batch inside of executor) is equal to one task?

That recursively uses "executor" and does not make much sense to me.

Every task produce only a partition?

Almost.

Every task produce an output of executing the code (it was created for) for the data in a partition.

The number of the Spark tasks equal to the number of the Spark partitions?

Almost.

The number of the Spark tasks in a single stage equals to the number of RDD partitions.