Let multiple requests wait for the first to complete (caching)

Question

I have a web api that has some get methods that raise some distributed queries to multiple systems that are quite long running and cpu expensive. So we added an output caching. Works like a charm and after the first query is processed it is quite fast.

So now i have a (maybe common?) problem when the cache is released. Imagine the scenario that the cache is invalidated or expired (for some good reasons) and not one client at a time call the web api but 100 clients. What currently happens is that 100 clients start to run the long running query and the cpu explodes to 100%, the server instances breaks down and does not respond anymore. What i basically want is that only the first requests does the processing and lets all other wait until he finished with the query and added the result to the cache. then all other ones can get the response from the cache.

I know there are around 1'000 ways to achieve it but i wonder if there is a general pattern or a best practice for this? Or do i have to implement locks? Or using TPL or what? Frankly speaking i have no clue where to start?

Thank you very much in advance.

Edit Beside the very good answer below i also found this article: http://www.asp.net/aspnet/overview/developing-apps-with-windows-azure/building-real-world-cloud-apps-with-windows-azure/queue-centric-work-pattern

Answer 1

Desired way is using message queues or a service bus. Enqueue your tasks and stop dequeuing them while building the cache.

This is easier to handle, because it's just about pausing a background service (for example, you can implement the message queue / service bus processing from a Windows service - you might want to take a look at Topshelf for this -) and resuming it.

Your API should enqueue tasks rather than processing them directly.

Possible message queues:

• MSMQ. Good for simple solutions. It comes with Windows out-of-the-box.
• RabbitMQ.
• Azure Service Bus. It can be also used in an on-premises basis with Azure Pack.
• NServiceBus

In your case I would go the MSMQ way, because it's a simple scenario where you just want to enqueue/dequeue messages...

What if I need it in the same process as Web API...

Matias thank you very much for your answer and the good proposal. What way would you propose if i am forced to go a inprocess way, so doing it in the same process as the web api is hosted?

A simple solution could be simulating a not that reliable queuing system using ConcurrentQueue<T> and creating a task processor thread which might be paused or resumed while the cache is rebuilt.

If you're hosting your WebAPI on IIS, I would switch to hosting it in a process, also called self-host. Learn about this reading this WebAPI OWIN/Katana self-host tutorial.