Implement Rest API with waiting list

Question

We have a rest API which some functions takes a few seconds to run. In order to be able to work properly, theses functions must lock some file resources, so while a file is locked, any other call to the function would result in an exception. It would be rather easy for us to simply check if the resource is available and once it is, take it, perform the task, and relase it. However, when this is done, there are no consideration about who called the function first so in a worst case scenario, the first person waiting could wait a lont time before getting access to it, which could result in a very long waiting time.

Instead, what I would like to do is use some sort of priority or wating list, where the first person in line will always be the first, kind of like those waiting lounge where to wait to buy tickets. I tried looking on the web but either it's not possible or I can't find the right words to look for it, so I am wondering if it has any implementation at all. However, english is not my first language so I don't know if I am looking at the right thing at all.

The actual running time of the functions are not pretty long and it would be surprising more than 3 persons at a time would actually need it so I doubt delay would be a problem, at least most of the time, but it's essential that they are not run at the same time. in the end, if it is not possible, I might end up with checking whether the file is in use and wait if it's not but I find this solution not "elegant" and would rather do it the best way possible.

Thanks

Answer 1

You might not find a specific Code solution to this because its implementation will be highly specific to your business requirements, skills, budget and the operating environment.

The elegant solution that you are looking for might cost more or take longer to implement than you can justify, in which case just fail the process and force the caller to wait and retry, which is probably where you are at right now.

There are different architectures available to solve issues like this but you can roll your own, lets break it down so you have some simpler concepts to research:

1. One caller at a time access to a REST based resource
2. FIFO Queue to identify callers and their priority
3. Mechanism to respond to the caller of a REST based resource asynchronously

As this is an API we can assume one key important fact, even if this is not specifically true for you now, you should design your API as if it will be or you will run into issues down the track:

A Web API will be deployed to redundant or highly available infrastructure.

Think cloud, server farms and load balancing... but even if you deploy to a single server running apache or IIS your API will usually be running as multiple processes.

• This means you can't use simple .NET threading concepts like lock, Mutex, Semaphore, async, await to manage single user access.

• It also means that you can't use .NET Queues to manage the priority, or anything from the System.Collections.Concurrent library because it will not have any knowledge or access across processes or servers.

• It also means you might not be able to rely on traditional file system locks if you are accessing physical files, unless the file are served from a cloud based or otherwise managed distributed file system.

• You should NOT try to implement this as a synchronous solution where the caller waits for a response from the web server. It can be done, but there are too many variables involved that you can't control to make this reliable, like how long the caller will wait before detecting this as a timeout.

The Process should be Asynchronous

I don't mean you should use async await code patterns, instead you should design the process such that there is a call to Begin the task, which should register your place in the queue, and then there is either a mechanism for the caller to check for task completion over time, or a way to call back to the originating process.

In a Web based API the call to Begin this critical task should immediately or synchronously return to the caller, the caller should poll or wait for a callback with the response.

Designs where the caller process polls for completion are simpler to implement from an API perspective, as it moves most of the decision logic on what to do next, out of the API.

• Callbacks could be in the form of a post back to a specific URL that the caller provides are also simple to implement, this is a standard pattern implemented supported by many credit card payment gateways. If the calling process supports WebHooks then this an implementation of the same concept.

• Web Sockets is a mechanism you can use to directly respond to the calling process, in .NET you can look at SignalR but you can do it from first principals.

  Many modern 3rd party products or services that solve issues like yours are based on Sockets as it is a more efficient pattern than the client polling for status updates when you want to achieve (near) real-time response.

• Push notifications is another implementation similar to WebSockets (part of the notification chain does use sockets) that you might be familiar with, you can use these notifications to call into client apps, even when the app has been closed on the client device...

  • well sort of, regardless of its validity in this situation, it is still an option and often can be very simple to implement.

Before you go down the direction of callbacks and sockets, consider if setting a flag in a record in the application database that makes the file available for the next caller or sending a simple email, SMS or push notification is enough.

The Queue needs to be external

For many tasks, it might be enough that the response to the Begin request (the lock aquisition) simply states that the file is in use by user x, and you should try again later. If the expected wait time is a small number of seconds or less, then the client process might not even show a notification to the user, it might just keep retrying.

Due to the above mentioned architecture constraints, even in this simple scenario it is important that the locking mechanism is a single point of truth stored out side of your executing code.

It could be a simple config or text file stored in a central location, it could be an entry in a database, you could any other kind of distributed cache, but it MUST be managed outside of your code process, so that multiple instances of your API can all access it.

You are correct in assuming we must rely on someone else having already solved this conundrum for us...

I would like to do is use some sort of priority or wating list, where the first person in line will always be the first, kind of like those waiting lounge where to wait to buy tickets

You are describing a FIFO (First In, First Out) Queue. There are products out there to simplify this, but the key is that your queue mechanism needs to be a single point of truth stored out side of your executing code. In this queue you should store the information (or a pointer to it) about the request, and how to respond when the task is completed.

What are the Options?

Whether you choose to roll your own, or you go with a 3rd party product or service, I would still encourage you to maintain your own record of the state of the task. If you have a database backend, then this might involve a new table to record the following information:

• who requested the task
• what task was requested
• when was the task requested
• what data does the task operate on
• what state is the task in
• what to do when the task is completed, or who to notify

every call to Begin the task should result in a new record in this table, this can simply be the execution log if nothing else, but ultimately it could form the basis for the response to the request.

the response should include some identifier or receipt number that could be used to lookup this queued task request entry.

The good thing about using a database is that it we can also use it to implement a simple queue. The bad thing is that this can become incredibly inefficient, lets look at it anyway, because it is very easy to get started with.

If your request pattern "might" be able to run synchronously in some situations and have to be queued in others, you should consider making it ALWAYS queued.

You can simplify both the API code and the client-side code if we force the pattern to be always asynchronous. It will reduce the branching logic that would be needed to handle the two cases, if the queue is empty we can kick off a manual execution.

Once you have requests going into the table that represents the makeshift queue you can now write a separate process to marshal the tasks, to ensure the one-at-a-time execution, from a database point of view this could be a simple pseudo query:

select the first Task
where the task is NOT in progress

select the oldest queued request 
for that task
where the task has not yet been completed

From a database point of view you would execute this query on a regular basis, to detect the next row to process.

Then you need another process to execute the task, this could be written as another endpoint in your API, but it should be able to be called entirely isolated from the original request handler. It might be easier to write it as a stand alone console app, or you write the marshalling or coordination logic into the console app and get it to call the correct endpoint when it detects a new task to complete.

At the start of the task, the code should update the queue table to indicate the task is in process. At the end of the task the code should also update the queue table to indicate that the task has completed.

You can't rely on your code to always update the database when or if it has completed, code can be aborted prematurely for all sorts of reasons, so you should also build some sort of timeout mechanism into your queue and processing state query logic.

This covers the one at a time, and the priority order to ensure the tasks do get completed, but it doesn't address call-back.

Calling back to the client, or notifying them that the task has been completed is really out of scope here, we've already covered some of the options, simply setting a flag on the queue record might be enough if your client can poll the queue as well to determine if the process it is waiting on has completed. From this client this might appear to be synchronous when in fact it is not.

Depending on your situation, you may not need a timing mechanism at all and can drive the processing from the client. First the client submits a call to put the task in the queue, it then repeatedly calls the processor endpoint which will either return a "waiting" response another task for the same record is pending (having been called from another client), or it will start the next queued task. When that response comes back the client checks if it's task was completed yet, if not it calls again.

Can you see how with the different working parts going on here and the different possible places to inject code to define, manage and execute this queue concept that it would be hard to find a single straight forward how to document?

Queues in Computing

This concept of a queue is not new by any means, in fact it is fundamental to our understanding of user interface design. In your user application your code will issue a sequence of requests to perform operations, although it is often abstracted away, you can choose for your request to be processed on the UI thread, a random available one, or a specific one. You can also specify different degrees of priority.

In many languages or default implementations will try to execute synchronously, but know that the operating system or the lower level kernel will still implement some sort queue to allow multiple applications and processes to appear to be operating concurrently. In Windows this is ultimately the message pump.

Cloud based Queues

There are many highly robust cloud based queue mechanisms available, offering high availability, FIFO and exactly once delivery options. With the evolution of IoT there are now a lot of production ready low cost implementation flavours that have direct support and code examples for .NET.

Disclaimer: I'm not in any way affiliated with MS, but Azure is my personal cloud vendor of choice, I assume that many other providers will offer the same or similar products and services but as I don't know them I haven't directly referenced them here

A simple place to start is Azure Queue Storage. This is an efficient pattern for managing the exactly one at a time and the FIFO aspects of your issue. If you had started with the previous advice to first split the workload into Request, Detect, Process, React chain of events then a cloud based queue can assist you with the Request and Detect stage, but it is really the marshalling/orchestration of the processing, so it replaces the previous

Ignore the Storage aspect of the name, it is a simple queue implementation, the name reflects both the technology stack and the licensing model through which the service is offered.

If you need more customisation options, or you end up with lots of queues or you move towards more concurrent processing then the Publish-Subscribe pattern offered by Azure Service Bus topics and subscriptions might be more appropriate.

How far you go down this path will depend on your budget, the frequency of tasks, the concurrency of processing and your ability to adapt these tools to your business needs.

Interesting services to investigate are Azure Event Hub and Event Grid, this is the MS guidance on how to choose between them - Event Grid being the latest offering which aims to simplify the code interactions and concepts to replicate the way we use events and event handlers in our programs, it means you don't need an intermediary processor host, client apps could effectively raise events in other clients... This answer on SO is a decent summary

---

What NOT to do

You have already listed some options that you have considered and discarded, the following table summarises common solutions and their suitability, X means it is supported, - means you might technically be able to make it work, but you probably shouldn't try, not in this type of environment ;)

Solution	Multiple Threads	Multiple Process	Multiple Server	Lock	Queue	Callback
.Net async await or (System.Threading)	X			X	-	X
.Net lock	X			X
.Net Queue or (System.Collections.Concurrent)	X				X
Traditional File System Lock	X	X	-	X
*Database Row Level Lock	X	X	X	X
Cloud or Distributed File System Lock	X	X	X	X

• Database Row Level Lock
  This is often seen as the silver bullet to these types of issues, as databases implement locking in a very reliable way, however the implementation often interrupts other program flow or at at least only solves the Lock issue, you would still need to separately manage the queue and callback.

Many of the above techniques can be deployed together to achieve your needs, but this problem cannot be solved by any one of them alone.