How does threading save time?

Question

I am learning threading in C#. However, I can't understand that which aspects of threads are actually improving performance.

Consider a scenario where there only a single core processor exists. Splitting your task into multiple threads uses the same process context (shared resource) and they run simultaneously. As the threads are just sharing time, how come their run time (turnaround time) is less than a single threaded process?

Answer 1

In a single core CPU the advantage that you get is through asynchrony. Using threads is one way of achieving that (although not the only way).

Imagine the process for cooking a meal. Which do you think is faster:

1. Start some water boiling. Wait for it to finish.
2. Add some noodles. Wait for them to finish being cooked.
3. Wash/prep some vegetables.
4. Stir fry the vegetables.
5. Put on plate and serve.

Or instead:

1. Start some water boiling.
2. While the water is boiling wash/prep some vegetables.
3. Add some noodles to the pot of boiling water.
4. Stir fry the vegetables while the noodles are being cooked.
5. Put on plate and serve.

From my experiences, the second is quicker.

The general idea here is that in many situations when programming you will have an operation that takes some time, but it doesn't require work from the CPU to be completed. A common example is IO. When you send a request off to the database to go get some information it's common for there to be other things for you to do while you wait for that request to come back. Perhaps you can send several requests and then wait for them to finish, rather than starting one, waiting on it, then starting the next, waiting, and so on (although sometimes you have to do the latter).

Now, if the work that you need to do is CPU bound work then you'll really only get benefits out of threading if you have multiple cores on your CPU such that work can actually be done in parallel, and not just asynchronously. For example, a lot of graphics related work (multiplying matrices, to give a simple example) often involves doing a lot of basic math. If you have several cores these operations often scale very well. If you don't have multiple cores (or a GPU, which is effectively a CPU with a lot of very small and simple cores) there isn't much point in using threads.

Answer 2

Consider a scenario where there only a single core processor exists. Splitting your task into multiple threads uses the same process context (shared resource) and they run simultaneously. As the threads are just sharing time, how come their run time (turnaround time) is less than a single threaded process?

You are entirely correct to be skeptical of any claimed speedup here.

First off, as Servy and others point out in their answers, if the jobs are not processor bound then clearly there can be some speedups here because while the processor is idle waiting for the disk or the network to come back, it could be doing the work of another thread.

But let's suppose you have two processor-bound tasks, a single processor, and either two threads or one thread. In the one-thread scenario it goes like this:

• Do 100% of the work of job 1. Suppose this takes 1000 ms.
• Do 100% of the work of job 2. Suppose this takes 1000 ms.

Total time: two seconds. Total jobs done: two. But here's the important bit: the client that was waiting for job 1 got their job done in only one second. The client that was waiting for job 2 had to wait two seconds.

Now if we have two threads and one CPU it goes like this:

• Do 10% of the work of job 1, for 100 ms.
• Do 10% of the work of job 2, for 100 ms.
• Do 10% of the work of job 1
• Do 10% of the work of job 2 ...

Again, total time two seconds, but this time the client that was waiting for job 1 got their job done in 1.9 seconds, nearly 100% slower than the one-thread scenario!

So that's the moral of the story here, that you are entirely correct to point out. If the following conditions are met:

• The jobs are CPU-bound
• There are more threads than CPUs
• The work is useful only for its end result

Then adding more threads only slows you down.

Libraries such as the Task Parallel Library are designed for this scenario; they try to figure out when adding more threads will make things worse, and try to only schedule as many threads as there are CPUs to serve them.

Now, if any of those conditions are not met then adding more threads is a good idea.

• If the jobs are not CPU bound then adding more threads allows the CPU to do work when it would otherwise be idle, waiting for network or disk.

• If there are idle CPUs then adding more threads allows those CPUs to be scheduled.

• If partially-computed results are useful then adding more threads improves the situation because there are more opportunities for clients to consume partially-computed results. In our second scenario, for instance, the clients of both jobs are getting partial results every 200 milliseconds, which is fair.