Multithreaded .NET RabbitMQ publisher

Question

We have the following scenario using the .NET RabbitMQ library:

A worker thread picks up 'request' messages from a queue and dispatches them on multiple worker threads for processing. When complete, each worker thread then sends another message.

My question is what 'pattern' do people recommend for the sender in order to get the best throughput and stability?. For example:

1) A singleton 'publisher' instance that is accessed by all worker threads with a single Connection and IModel (using a 'lock' to synchronize access to the IModel)

2) A singleton 'publisher' instance that is accessed by all worker threads with a single Connection and which creates a new IModel for each send request.

or something else?

Answer 1

According to RabbitMQ user guide "IModel instances should not be used by more than one thread simultaneously: application code should maintain a clear notion of thread ownership for IModel instances" In case a IModel is shared a lock should be use as you said but in my opinion this leads to a more complex code because connection and model should be keep alive in case of disconnection.

Although you don't mention if transactions are required in order to achieve more reliability on message delivery the channel must be set on transaction mode and maybe you will need a transaction per delivery.

Using a new model per delivery ensures a more simple error management but it obviously will slow down the throughput (and the same by using transactions).

You could also, depending on your requirements, use non durable queues and direct or fanout exchanges which provide better throughput.

As an example, on a development machine a windows service that consumes messages from a queue using one single thread (deserializing, making some business logic, and finally send a new serialized message using transactions by opening/closing connection and model can achieve about 3000 messages processed per second. (serialization was done via XmlSerializer which is worst than DataContractSerializer)