MSMQ messages always arrive delayed by exactly 3 minutes on the same machine

Question

I'm facing an extremely puzzling problem. I have a Windows service that monitors two MSMQ queues for input and sends messages to another MSMQ queue. Although the send operation seems instant from the service's perspective it actually takes the message exactly three (3) minutes to arrive (as shown in the properties window in the MSMQ MMC). I've been testing this problem with nothing else listening on the other side so that I can see the messages piling up. This is how the service sends messages:

var proxyFactory = new ChannelFactory<IOtherServerInterface>(new NetMsmqBinding(NetMsmqSecurityMode.None)
{
    Durable = true,
    TimeToLive = new TimeSpan(1, 0, 0),
    ReceiveTimeout = TimeSpan.MaxValue
});

IOtherServerInterface server = this.proxyFactory.CreateChannel(new EndpointAddress("net.msmq://localhost/private/myqueue"));

var task = new MyTask() { ... };
using (TransactionScope scope = new TransactionScope(TransactionScopeOption.Required))
{
    server.QueueFile(task);
    scope.Complete();
}

The service is running on Windows Server 2008 R2. I also tested it on R1 and noticed the same behavior. Again, everything happens on the same machine. All components are deployed there so I don't think it could be a network issue.

EDIT #1:

I turned on the WCF diagnostics and what I noticed is very strange. The MSMQ datagram does get written normally. However, after the "a message was closed" trace message there is nothing going on. It is as if the service is waiting for something to happen. Exactly 3 minutes later and exactly when the MSMQ message arrives (according to the MSMQ MMC), I see another trace message about a previous activity. I suspect there is some kind of interference.

Let me give you more details about how the services work. There is an IIS app which receives tasks from clients and drops them in an MSMQ queue. From there, the troublesome service (MainService) picks them up and starts processing them. In some cases, another service (AuxService) is required to complete the task so MainService sends a message (that always gets delayed) to AuxService. AuxService has its own inbox queue where it receives MSMQ messages and when it's done, it sends an MSMQ message to MainService. In the meanwhile, the thread that sent the message to AuxService waits until it gets a signal or until it times out. There is a special queue where MainService looks for messages from AuxServices. When a message is received the abovementioned thread is woken up and resumes its activity.

Here's a representation of the whole architecture:

• IIS app -> Q1 -> MainService
• MainService -> Q2 -> AuxService
• AuxService -> Q3 -> MainService

Although all operations are marked with OneWay, I'm wondering whether starting a MSMQ operation from within another MSMQ operation is somehow illegal. It seems to be the case given the empirical evidence. If so, is there away to change this behavior?

EDIT #2:

Alright, after some more digging it seems WCF is the culprit. I switched both the client code in MainService and the server code in AuxService to use MSMQ SDK directly and it works as expected. The 3 minute timeout I was experiencing was actually the time after which MainService gave up and considered that AuxService failed. Therefore, it seems that for some reason WCF refuses to perform the send until the current WCF activity exits.

Is this by design or is it a bug? Can this behavior be controlled?

Answer 1

You have transactions setup on the queue code, do you have the msmq object setup for transactions? 3 minutes sounds like the timeout period for a Distributed Transaction Coordinator enlistment.