Reading from MSMQ slows down when there is a lot of messages queued

Question

Short introduction

I have a SEDA based system, and used MSMQ for communication (event triggering) between the different applications/services.

One of these services gets messages by file, so I have a file listener that reads the file content and inserts this into a queue (or actually 4 different queues, but that's not very important for the first question).

Server is Windows Server 2008

First question - read slows down

My application that reads these messages at the other side normally reads about 20 messages from the queue per second, but when the service that posts messages start queuing some thousand messages, the read goes down, and the read application only reads 2-4 messages per second. When there is no posting to the queue, the read application can again read up to 20 messages per second.

The code in the reading application is pretty simple, developed in C#, I use the Read(TimeSpan timeout) function in System.Messaging.

Q: Why does the read slows down when there is a lot of messages posted to the queue?

Second question - limitations of TPS

An additional question is about the read itself. It seems there is no difference in how many messages I can read per second if I use 1 or 5 threads to read from the queue. I've also tried implementing a "round robin solution" where the post service are posting to a random set of 4 queues, and the read application had one thread listening to each of these queues, but there is still only 20 TPS even if I read from 1 queue with 1 thread, 1 queue with 4 threads or 4 queues (with one thread per queue).

I know the processing in the thread takes about 50 ms, so 20 TPS is quite correct if there is only one message processed at the time, but the clue with multi threading should be that messages are handled in parallel and not sequential.

There is about 110 different queues on the server.

Q: Why can't I get more than 20 messages out of my queue at the time even with multi threading and the use of several queues?

This is the code running today:

// There are 4 BackgroundWorkers running this function
void bw_DoWork(object sender, DoWorkEventArgs e) 
{
    using(var mq = new MessageQueue(".\\content"))
    {
        mq.Formatter = new BinaryMessageFormatter();

        // ShouldIRun is a bool set to false by OnStop()
        while(ShouldIRun)
        {
            try
            {
                using(var msg = mq.Receive(new TimeSpan(0,0,2))
                {
                    ProcessMessageBody(msg.Body); // This takes 50 ms to complete
                }
            }
            catch(MessageQueueException mqe)
            {
               // This occurs every time TimeSpan in Receive() is reached
               if(mqe.MessageQueueErrorCode == MessageQueueErrorCode.IOTimeout) 
                   continue;
            }
        }
    }

But even if there are 4 threads, it seems all waits for the function to enter the "Receive" point again. I've also tried using 4 different queues (content1, content2, content3 and content4), but still i get 1 message processed every 50 ms.

Does this have anything to do with the TimeSpan in Receive(), and/or is it possible to omit this?

Another question is if the use of private queues, instad of public will solve anything?

Answer 1

Performance issues.
You don't mention if all the code is running on the server or if you have clients remotely accessing the queues on the server. From the speed, I'll assume the latter.
Also, are the queues transactional?
How large are the messages?

If you want to read a message from a queue, your application does not connect to the queue itself. Everything goes between the local queue manager and the remote queue manager. The queue manager is the only process that writes to, and reads from queues. Therefore having multiple queues or a single queue won't necessarily perform any differently.

The MSMQ queue manager is therefore going to be a bottleneck at some point as there is only so much work it can do at the same time. Your first question shows this - when you put a high load on the queue manager putting messages IN, your ability to take messages OUT slows down. I'd recommend looking at performance monitor to see if MQSVC.EXE is maxed out, for example.