Nested transactions in StatefulService to save async state of aborted transaction

Question

I have a ReliableQueue<MyTask> which is enqueued into in a different scope, and I'm dequeuing tasks in a transaction, then want to run some long running calculations on each task.

The problem here is that in case my queue transaction is aborted, I don't want to lose the instance of the long calculation. It will keep running in the background, independent, and I just want to check if its completed or not once I retry to process the tasks.

Code segment:

public void protected override async Task RunAsync(CancellationToken cancellationToken)
{
    var queue = await StateManager.GetOrAddAsync<IReliableQueue<MyTask>>(...);
    while(!cancellationToken.IsCancellationRequested)
    {
        using (var transaction = ...)
        {
            var myTaskConditional = await queue.TryDequeueAsync(transaction);
            if (!myTaskConditional.HasValue)
            {
                break;
            }
            await DoLongProcessing(myTaskConditional)
            await transaction.CommitAsync();
        }
    }
}

private async void DoLongProcessing(MyTask myTask) {
    var dict = await StateManager.GetOrAddAsync<IReliableDictionary<Guid,Guid>>(...);
    Conditional<Guid> guidConditional;
    using (var transaction = ...)
    {
        guidConditional = await dict.TryGetValueAsync(myTask.TaskGuid);
        if (guidConditional.HasValue) {
            await transaction.CommitAsync();
            // continue handling knowing we already started, continue to wait for 
            await WaitForClaulcationFinish(guidConditional.Value);
        }
        else {
            // start handling knowing we never handled this task, create new guid and store it in dict
            var runGuid = await StartRunningCalculation(runGuid);
            await dict.AddAsync(myTask.TaskGuid, runGuid);
            await transaction.CommitAsync();
            await WaitForClaulcationFinish(runGuid);
        }
    }
}

My concern: I'm using nested transactions and that's not recommended.

Is there actually a risk of deadlock here if I'm using the transactions solely for the ReliableQueue or ReliableDictionary separately?

Is there a better intended design for what I'm trying to achieve?

Answer 1

You should not be doing anything long running within a transaction. Take a look at the priority queue service I published. Take the item out of the queue an place it into a collections while doing work, then when done, either put it back into the queue or complete the work.