how to design threading for many short tasks

Question

I want to use multi-threads to accelerate my program, but not sure which way is optimal.

Say we have 10000 small tasks, it takes maybe only 0.1s to finish one of them. Now I have a CPU with 12 cores and I want to use 12 threads to make it faster.

So far as I know, there are two ways:

1.Tasks Pool

There are always 12 threads running, each of them get one new task from the tasks pool after it finished its current work.

2.Separate Tasks

By separating the 10000 tasks into 12 parts and each thread works on one part.

The problem is, if I use tasks pool it is a waste of time for lock/unlock when multiple threads try to access the tasks pool. But the 2nd way is not ideal because some of the threads finish early, the total time depends on the slowest thread.

I am wondering how you deal with this kind of work and any other best way to do it? Thank you.

EDIT: Please note that the number 10000 is just for example, in practice, it may be 1e8 or more tasks and 0.1 per task is also an average time.

EDIT2: Thanks for all your answers :] It is good to know kinds of options.

Answer 1

So one midway between the two approaches is to break into say 100 batches of 100 tasks each and let the a core pick a batch of 100 tasks at a time from the task pool.

Perhaps if you model the randomness in execution time in a single core for a single task, and get an estimate of mutex locking time, you might be able to find an optimal batch size.

But without too much work we at least have the following lemma :

The slowest thread can only take at max 100*.1 = 10s more than others.