Optimistic Caching Concurrency Design Pattern

Question

I have a web service that is running on a cluster of servers. This web service does some internal processing, and then may make a call out to an external service that incurs a fee.

I want to put in some caching so that if I receive identical requests to the service (which is guaranteed to happen), then I do not have to repeat the processing, saving both processing time/power and also the cost incurred in the external part of the service call.

However, I am struggling to figure out how to manage this caching when I have the following constraints

• The service is running on multiple web servers for High Availability and scalability
• The request may take up to 5 seconds to respond, but in the mean time, I may have received 2 or 3 other identical requests.

How can I hold off on executing the other service calls, until the first one has responded (therefore available in the cache), when working in a distributed environment.

I have thought about putting in a front-proxy pattern and building up a queue of identical requests within the proxy, so that when the first returns, it can also return the same response to the others. Is this the correct pattern, or is there a better concurrency pattern that deals with this scenario?

Answer 1

You could

1. compute a cryptographic hash of the request
2. see if the result already is in the database for this hash, and if so, return it
3. store the hash in database with a "result pending" status
4. call the web service and update the row in the database with the result.

At step 2, if the hash already is in the database, with the "result pending" status, you could poll the database every X milliseconds and finally return the result once it's there.

The devil is in the details, of course, because you would have to decide what you do in case an error occurs:

• do you return an error for all the subsequent identical requests?
• do you cause the waiting threads to retry calling the web service?
• do you return an error, but only for some time, and then retry?