Service Fabric reliable queue long operation

Question

I'm trying to understand some best practices for service fabric.

If I have a queue that is added to by a web service or some other mechanism and a back end task to process that queue what is the best approach to handle long running operations in the background.

1. Use TryPeekAsync in one transaction, process and then if successful use TryDequeueAsync to finally dequeue.
2. Use TryDequeueAsync to remove an item, put it into a dictionary and then remove from the dictionary when complete. On startup of the service, check the dictionary for anything pending before the queue.

Both ways feel slightly wrong, but I can't work out if there is a better way.

Answer 1

One option is to process the queue in RunAsync, something along the lines of this:

protected override async Task RunAsync(CancellationToken cancellationToken)
{
    var store = await StateManager.GetOrAddAsync<IReliableQueue<T>>("MyStore").ConfigureAwait(false);
    while (!cancellationToken.IsCancellationRequested)
    {
        using (var tx = StateManager.CreateTransaction())
        {
            var itemFromQueue = await store.TryDequeueAsync(tx).ConfigureAwait(false);
            if (!itemFromQueue.HasValue)
            {
                await Task.Delay(TimeSpan.FromSeconds(1), cancellationToken).ConfigureAwait(false);
                continue;
            }

            // Process item here
            // Remmber to clone the dequeued item if it is a custom type and you are going to mutate it.
            // If success, await tx.CommitAsync();
            // If failure to process, either let it run out of the Using transaction scope, or call tx.Abort();
        }
    }
}

Regarding the comment about cloning the dequeued item if you are to mutate it, look under the "Recommendations" part here: https://azure.microsoft.com/en-us/documentation/articles/service-fabric-reliable-services-reliable-collections/

One limitation with Reliable Collections (both Queue and Dictionary), is that you only have parallelism of 1 per partition. So for high activity queues it might not be the best solution. This might be the issue you're running into.

What we've been doing is to use ReliableQueues for situations where the write amount is very low. For higher throughput queues, where we need durability and scale, we're using ServiceBus Topics. That also gives us the advantage that if a service was Stateful only due to to having the ReliableQueue, it can now be made stateless. Though this adds a dependency to a 3rd party service (in this case ServiceBus), and that might not be an option for you.

Another option would be to create a durable pub/sub implementation to act as the queue. I've done tests before with using actors for this, and it seemed to be a viable option, without spending too much time on it, since we didn't have any issues depending on ServiceBus. Here is another SO about that Pub/sub pattern in Azure Service Fabric

Answer 2

If very slow use 2 queues.. One a fast one where you store the work without interruptions and a slow one to process it. RunAsync is used to move messages from the fast to the slow.