Azure Service Bus keeps throwing MessageLockLostExceptions

Question

I keep getting MessageLockLostExceptions when processing messages.

• I am using Microsoft.Azure.ServiceBus 3.2.0 with .NET Core 2.1.
• I have a queue with LockDuration set to 30 seconds already containing a number of messages to be processed.
• I took the very basic message-receiving tutorial code from https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dotnet-get-started-with-queues#receive-messages-from-the-queue, which works fine.

Now I want to simulate a slightly longer running message processing task (but still well within LockDuration) by adding Task.Delay(10_000). But then I get a MessageLockLostException for exactly every 4th message.

This happens even if I set MaxAutoRenewDuration = TimeSpan.FromDays(30) and PrefetchCount = 0.

---

This is the message processing method, which I changed slightly to print out the remaining lock duration:

    private static async Task processMessagesAsync(Message message, CancellationToken token)
    {
        Console.Write($"Received message: {message.SystemProperties.SequenceNumber}. Remaining lock duration: {message.SystemProperties.LockedUntilUtc - DateTime.UtcNow}");
        await Task.Delay(10000);
        await queueClient.CompleteAsync(message.SystemProperties.LockToken);
        Console.WriteLine(" - Complete!");
    }

---

Sample output:

======================================================
Press ENTER key to exit after receiving all the messages.
======================================================
Received message: 3659174697238584. Remaining lock duration: 00:00:30.8269132 - Complete!
Received message: 19421773393035331. Remaining lock duration: 00:00:20.5271654 - Complete!
Received message: 11540474045136941. Remaining lock duration: 00:00:10.3372697 - Complete!
Received message: 15762598695796784. Remaining lock duration: 00:00:00.1776760
Message handler encountered an exception     Microsoft.Azure.ServiceBus.MessageLockLostException: The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue. Reference:2c6caac3-e607-4130-a522-f75e4636e130, TrackingId:3ff82738-664d-4aca-b55f-ba3900f1c640_B17, SystemTracker:ocgtesting:queue:workflow~63, Timestamp:2018-12-12T17:01:59
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.OnRenewLockAsync(String lockToken) in C:\source\azure-service-bus-dotnet\src\Microsoft.Azure.ServiceBus\Core\MessageReceiver.cs:line 1260
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.<>c__DisplayClass74_0.<<RenewLockAsync>b__0>d.MoveNext() in C:\source\azure-service-bus-dotnet\src\Microsoft.Azure.ServiceBus\Core\MessageReceiver.cs:line 771
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func`1 operation, TimeSpan operationTimeout) in C:\source\azure-service-bus-dotnet\src\Microsoft.Azure.ServiceBus\RetryPolicy.cs:line 83
at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func`1 operation, TimeSpan operationTimeout) in C:\source\azure-service-bus-dotnet\src\Microsoft.Azure.ServiceBus\RetryPolicy.cs:line 105
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.RenewLockAsync(String lockToken) in C:\source\azure-service-bus-dotnet\src\Microsoft.Azure.ServiceBus\Core\MessageReceiver.cs:line 773
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.RenewLockAsync(Message message) in C:\source\azure-service-bus-dotnet\src\Microsoft.Azure.ServiceBus\Core\MessageReceiver.cs:line 742
at Microsoft.Azure.ServiceBus.MessageReceivePump.RenewMessageLockTask(Message message, CancellationToken renewLockCancellationToken) in C:\source\azure-service-bus-dotnet\src\Microsoft.Azure.ServiceBus\MessageReceivePump.cs:line 248.

---

Full code here: https://pastebin.com/sFGBgE0s

Answer 1

One thing that is missing from your repro is the queue description. It's important to note such details as the problem you're experiencing has nothing to do with the client and is most likely either related to the broker or the underlying AMQP library.

For non-partitioned queues this setup works fine. It doesn't for partitioned queues (Standard tier). Can be observed with both the old and the new clients. I've raised a broker related issue for Azure Service Bus team to investigate.